This invention relates to the digital processing of video data to be displayed on a television screen and more particularly, to the processing of a subpicture.
Almost all televisions manufactured today are capable of interfacing with different sources of program materials, for example, a VCR, a digital versatile or video disk (xe2x80x9cDVDxe2x80x9d) player, cable, DSS, etc., that provide audio signals for creating sounds and associated video input signals for creating screen displays. Some of those sources provide digital audio and video input signals in accordance with the Moving Picture Expert Group MPEG-2 audio/video digital compression standard. Further, most televisions and/or their plug compatible program sources have user interactive capabilities with which a user may choose to have the programmed source provide subpicture data for providing a display of captions, subtitles, karaoke or simple animation on the screen along with the program material. Thus, contemporary televisions and/or DVD systems preferably have the capability of processing compressed digital input signals representing audio, video and subpicture data and providing digital output signals representing the desired sound, video and subpicture images. Most often, those digital output signals are converted to analog signals for use by known analog television display units.
The implementation of digital signal processing for providing an audio, video and subpicture display from an audio-video source of programmed material presents numerous design challenges that were not encountered in the prior processing of analog audio and video signals. For example, with digital signal processing, the audio, video and subpicture signals are separated and are processed independently. However, the playback of the audio, video and subpicture must be synchronized, so that there is a coordinated and coherent reproduction of the desired audio, video and subpicture provided from the source of program material.
The source, for example, a video disk, preferably provides the audio and video data in respective data packets in an xe2x80x9cMPEG-2xe2x80x9d format. Each of the audio, video and subpicture data packets is received from the source of video material in a continuous data stream. Each packet of video data includes a header block followed by a data block. The data block may include any number, for example one to twenty, of frames of video data that may include a full field of video data or be a coded group of pictures that includes its own header block identifying the picture type and display order. The header block for a video data packet includes control information, for example, the identity of the format of the video data, the type of compression, if used, picture size, display order, and other global parameters.
The audio data packet has a header block that again identifies the format of the audio data with instructions relating to how the audio data is to be decoded and processed to provide desired enhancements, if applicable. Following the header block, the audio data packet includes an audio data block that has any number of blocks or frames of audio data, for example, from one to approximately twenty blocks.
The subpicture data may be provided in a data packet in one of several formats. For purposes of this description, it will be assumed that the subpicture data is being provided in a Subpicture format that is defined by the known DVD standard. The Subpicture format includes a header block, a pixel data block, and a display control sequence (xe2x80x9cDCSQxe2x80x9d) command data block. Generally, the header is used to identify the general nature of the data, for example, the size of the subpicture unit and the location of the DCSQ commands. In the Subpicture format, the pixel data represents color and contrast information and is compressed using known compression techniques, for example, run length compression. The DCSQ command block includes one or more sets of commands that determine color and contrast globally within the subpicture. In addition, DCSQ command data may optionally include a Change Color-Contrast (xe2x80x9cCHG_COLCONxe2x80x9d) command which functions to change the color and contrast within the subpicture on a pixel by pixel basis.
Selected ones of the header blocks of the audio and video data packets and all of the header blocks of the subpicture data packets include a presentation time stamp (xe2x80x9cPTSxe2x80x9d) value which is a time stamp that is applicable to that data packet. The PTS value is a time reference to a system time clock that was running during the creation or recording of the audio and video data. A similar system time clock (xe2x80x9cSTCxe2x80x9d) is also running during the playback of the audio and video data, and if the audio, video and subpicture data are played back at the times represented by their presentation time stamps, the audio, video and subpicture data will be presented to the user in the desired synchronized manner. Therefore, the PTS value is used to synchronize the presentation or playback of the audio, video and subpicture data.
During the decoding of the audio data, it normally must be decompressed, reconstructed and enhanced in a manner consistent with the source of program material and the capabilities of the sound reproduction system. In some applications, audio data packets may contain up to six channels of raw audio data. Depending on the number of channels the sound reproduction systems can reproduce, for example, from two to six, the sound reproduction system selectively uses the channels of raw audio data to provide a number of channels of audio which are then stored in an audio FIFO.
The decoding of the video data normally requires decompression, conversion of partial frames into full frames and the recognition of full frames. The decoding of subpicture data requires the decompression of run length compressed bit maps of subpicture data. Simultaneously with the decoding process, audio, video and subpicture data is being played back to the user, and in that playback, the frames of audio and video data are being output and the subpicture is overlaid on top of the program video and the reconstructed audio, video and subpicture must be synchronized in the playback process such that the audio, video and subpicture present a coordinated and coherent presentation.
As will be appreciated from the foregoing, demultiplexing the audio, video and subpicture data packets is a complex process of deconstructing the data packets and storing the necessary decoding instructions as well as the content data itself to permit the decoding and playback of the data in a synchronized manner. One such process, is described in a copending U.S. patent application Ser. No. 08/901,090 entitled Method and Apparatus for Audio-Video Synchronizing, filed on Jul. 28, 1997, now U.S. Pat. No. 5,959,684 and assigned to the assignee of the present application. U.S. patent application Ser. No. 08/901,090 is in its entirety hereby incorporated by reference.
The interactive nature of current entertainment equipment presents additional problems in a synchronized playback of audio, video and subpicture data. For example, subtitles in a language different from the language of the audio are often displayed in the playback of programmed media, for example, a movie on a video disk. A video disk may contain subtitles in up to 32 different languages, and each subtitle language is stored as a subpicture data stream on the video disk. The user has the capability of switching between any of those 32 different languages at any point in time. Switching between different subpicture data streams often results in a temporary distortion in the subpicture display during the switching transition or a temporary loss of synchronization with the video being played.
Consequently, in a video system providing multiple streams of subpicture data, there is a need to provide an improved switching between different streams of subpicture data in response to randomly provided user requests.
The present invention provides a method and apparatus of a real time smooth transition between different streams of subpicture data without any delay or interference in the presentation of the main audio and video. The present invention has an advantage of utilizing only complete subpicture units in making a transition between first and second subpicture data streams. Thus, there is no partial subpicture displays that may appear as video noise during the transition, and there is no interruption to the presentation of the main audio ad video. The present invention utilizes all valid subpicture units in the first subpicture data stream prior to switching to the subpicture units in the second subpicture data stream, thus playing back the optimum number of multiplexed subpicture units in response to random user requests to change from one subpicture data stream to another and coordinating to the greatest extent possible the subpicture display to the video being played.
In accordance with the principles of the present invention and the described embodiments, the invention provides a method of operating a digital video processor to continuously demultiplex, store and decode audio and video data streams and play back corresponding frames of audio and video. The processor further continuously demultiplexes, stores and decodes a first subpicture data stream and plays back first subpicture data in synchronization with the playback of frames of video data. PTS values associated with SPU""s in the first subpicture data stream are continuously stored. The processor then detects a command instructing a transition from the first subpicture data stream to a second subpicture data stream and stores a PTS value associated with a first complete SPU of the second subpicture data stream so that it follows a PTS value associated with a last complete SPU in the first subpicture data stream. The processor then continuously stores PTS values associated with the SPU""s in the second subpicture data stream. The first subpicture data stream is played back in synchronization with corresponding frames of video according to the stored PTS values associated with the SPU""s of the first subpicture data stream; and the second subpicture data stream is played back in synchronization with corresponding frames of video according to the stored PTS values associated with the SPU""s of the second subpicture data stream. Thus, processor provides a transition from the first to the second subpicture data streams without playing back any partial SPU""s in either of the first or the second subpicture data streams and without introducing any delay or interference in the presentation of the main audio and video.
In another embodiment of the invention, the above processor determines a PTS value of a first complete SPU in the second subpicture data stream and successively compares the PTS value of the first complete SPU in the second subpicture data stream to the stored PTS values of the complete SPU""s in the first subpicture data stream beginning with a most recently stored PTS value. The processor then identifies a closest stored PTS value of the complete SPU""s in the first subpicture data stream that immediately precedes the PTS value of the first complete SPU in the second subpicture data stream. Thus, the processor provides a smoother transition from the first to the second subpicture data streams by playing back all possible SPU""s in the first subpicture data stream.
These and other objects and advantages of the present invention will become more readily apparent during the following detailed description taken in conjunction with the drawings herein.